Sequential testing over multiple stages and 
performance analysis of data fusion 



Gaurav Thakur* 
February 18, 2013 

Abstract 

We describe a methodology for modeling the performance of decision-level data fusion between 
different sensor configurations, implemented as part of the JIEDDO Analytic Decision Engine (JADE). 
We first discuss a Bayesian network formulation of classical probabilistic data fusion, which allows 
elementary fusion structures to be stacked and analyzed efficiently. We then present an extension of 
the Wald sequential test for combining the outputs of the Bayesian network over time. We discuss an 
algorithm to compute its performance statistics and illustrate the approach on some examples. This 
variant of the sequential test involves multiple, distinct stages, where the evidence accumulated from 
each stage is carried over into the next one, and is motivated by a need to keep certain sensors in the 
network inactive unless triggered by other sensors. 
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tion (JIEDDO) for supporting and funding this work, Dr. Tom Stark of JIEDDO for technical and opera- 
tional guidance, and Dr. Dave Colella and Dr. Garry Jacyna of MITRE for providing valuable feedback and 
suggestions on the paper. 

1 Introduction 

The JIEDDO Analytic Decision Engine (JADE) is a flexible software toolkit for studying the performance 
of sensor configurations for the detection of person-borne explosive compounds and other threat substances. 
JADE is designed to enable performance and tradeoff analyses between different, user-specified scenarios 
with given sensor placements and data fusion networks. JADE contains fundamental physics-based models 
of several sensor technologies of interest, such as nonlinear acoustic and radar-based detectors, along with 
a data fusion system that we focus on in this paper. The fusion system consists of a static component that 
combines the decisions of individual sensors at a fixed point in time, and a dynamic, time-dependent com- 
ponent that in turn fuses the outputs of the static structure at different times. The static component is based 
on a probabilistic graphical model, or Bayesian network, and accepts probability matrices from the physics- 
based sensor models as inputs (the details of which are abstracted from the fusion system). Its outputs are 
fed into the dynamic fusion framework, which is based on sequential hypothesis testing and produces per- 
formance metrics for the entire, fused sensor configuration. The purpose of the system is to determine the 
performance of a given fusion structure, as opposed to doing fusion on actual measurements. 
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We first discuss the static framework in Section |2] which allows elementary fusion structures to be stacked 
and analyzed efficiently. This material is fairly standard but serves as a background for the rest of the 
paper. We then describe an extension of the Wald sequential test in Section [3] that involves multiple, distinct 
stages, where the evidence accumulated from each stage is carried over into the next one. We show how the 
performance characteristics and decision times of such a test can be computed efficiently for time-dependent 
statistics and illustrate this approach on examples in Section@] This setup models a bank of anomaly sensors 
that observe a moving target over time, reach an initial fused decision, and if justified, activate additional 
sensors that continue to collect static fused evidence over time until a final decision is made about the target. 
The multiple-stage configuration allows sensors that have a high cost of operation to remain inactive unless 
specifically called upon. 

2 Static fusion using Bayesian networks 

The static fusion structure is formulated as a Bayesian network, i.e. a directed acyclic graph with each vertex 
representing a random variable and edges describing dependencies between the variables. A Bayesian net- 
work has the defining property that every vertex is conditionally independent of its ancestor vertices given 
its immediate parent vertices fij. Such networks are an intuitive framework for performing probabilistic 
inference among interconnected events in many different contexts, and are well suited for formulating a 
sensor fusion system. The vertices in our network represent the object, sensors and fusion centers. 

Suppose we have N sensors to be fused, each of which outputs hard decisions between M possibilities 
(with the first one corresponding to the case where no threat is present). Let H be the true object (or 
the hypothesis in a Bayesian setting) and S be the local decision of a given sensor. The performance of 
the sensor is described by the M x M matrix {P(S = m\H = m')}\< m ^ m i<M, which we write concisely as 
P(S = - \H). In this paper, we will generally focus on M = 2, corresponding to binary decision-level fusion, 
but the discussion in this section applies to other M as well. At any fusion center F with V parent vertices 
{S n }i<n<v> we can describe the fusion rule by the V-dimensional tensor P(F = •|{5 n }i< n <y), which for 
deterministic fusion rules consists only of and 1 elements. The performance of the entire system is given 
by P(D = - \H), where H is the root vertex in the graph and the system's final decision D is the last child 
vertex. This formulation enables the graph to take on essentially any desired form and allows different 
combinations of fusion centers and sensors to be stacked together, subject to the following rules that ensure 
that the fusion structure is meaningful. 

• Each sensor vertex must have the object and at most one fusion center as its parent. 

• At least one sensor must have only the object as its parent. 

• Each fusion center can have any combination of sensors and/or fusion centers as parents, as long as 
no cycles are formed in the graph. 

• No fusion center can have the object as a parent. 

• There must be exactly one fusion center with no children, representing the final decision. 

These rules ensure that all sensors in the graph observe the object and that all intermediate decisions are 
ultimately combined at a single, final fusion center. An example fusion network of this type is shown in 
Figure 12.11 

To determine P(D = -\H), we choose small subgraphs of the Bayes network at a time, each containing one 
fusion center and all its parent vertices, and marginalize over them using the standard method of belief 
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propagation (3]|4]|. This is done iteratively for each fusion center in parent-to-child order until all the fusion 
centers have been covered. For certain fusion rules, the probability matrix at any child fusion centers may 
depend on the outputs of any parent fusion centers, so this iterative procedure is much more simple and 
efficient than having the child fusion centers' conditional probabilities account for this dependence and 
computing marginal probabilities over the entire Bayes network at once. 



At each fusion center, JADE allows the user to choose between five elementary hard-decision fusion rules: 
the "and," "or," majority, Neyman-Pearson optimal and Bayes optimal rules [6 ]. Any of the five fusion rules 
can be used in the decision-level case of M = 2, while for M > 2, only the Bayes and majority rules are 
meaningful. At any given fusion center, let {S n }\< n <v be the local decisions of V sensors feeding into it, 
with < S n < M — 1, and let F be the fused decision. The "and" rule simply chooses F = 1 if all the S n = 1, 
and F = otherwise. Similarly, the "or" rule chooses F = if all the S n = 0, and F = 1 otherwise. It is 
clear that the "and" rule minimizes Pp while keeping Pp> > and the "or" rule maximizes Pp, while keeping 
Pp < 1 , so they can be thought of respectively as the least and most sensitive fusion rules available. The 
majority rule takes a majority vote between the sensors, i.e. F = moded^}), with a random, uniformly 
distributed decision taken if there is a tie between multiple choices. This is the only rule where the fused 
decision is potentially random. These three rules generally do not satisfy any good optimality criteria, but 
are conceptually simple and useful as a baseline for comparison against the two optimal rules. 

At a given fusion vertex in the network, the Neyman-Pearson rule (for M = 2) has the user specify a target 
false alarm probability P' F , and the system chooses the (deterministic) fusion rule that maximizes the local 
Pp, at that vertex, subject to the constraint Pp < P' F . The optimal rule is found by computing the likeli- 
hood ratios Y\n=i p\s'\h=o) ^ or ever y combination of individual sensor decisions {S n } and arranging them 






Figure 2. 1 : An example static fusion network. 
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in increasing order. The combinations are then partitioned into two subsets / and J such that for {S n } € /, 
Ul=iH s n\H = 0) < P' p and for {S n } G J, YZ=i p ( s n\ H = 0) > P' F . The solution is given by the rule that 
chooses F = for {S„} G / and F = 1 for {£„} G 7. 

For the Bayes fusion rule [fl] [2], the user specifies the costs of a false alarm Cp, a missed detection Cm 
and (for M > 2) a mix-up between two threat possibilities Cx- For each combination of individual sensor 
decisions {S n }, the system finds a fused decision F that minimizes the Bayes risk, or the expected cost of a 
wrong decision, 

m v 
F = argmi ni < 7 .< M £ C jM P{H =k— 1) n p ( 5 «l# = *~ 1), 

fc=l n=\ 

where C; ; = 0, Cy i = Cf and C\j = Cm for j > 2, and C ;J t = Cx for all other (,/',£). The optimal fusion 
rule can be found by simply looking at every combination {S n } individually and taking the best of the M 
possible fused decisions for each one. In general, finding this fusion rule is a computationally difficult dis- 
crete optimization problem, but this simple, brute-force approach is fast as long as V and M are fairly small 
(e.g. less than 10), as is the case in our scenarios of practical interest. 



3 Dynamic fusion using multiple-stage sequential testing 

Suppose now that we have two static fusion networks of sensors, each represented by a Bayesian network 
of the type described in Section Q] The sensors collect measurements from a target moving along a spec- 
ified path, with the static network producing a fused decision at every point in time based on the sensors' 
individual probabilities at each position. These fused decisions can be combined over time using Wald's 
theory of sequential probability testing. The classical Wald sequential test is essentially a one-dimensional 
random walk where the total likelihood ratio of the system makes "steps" in either direction, corresponding 
to different incoming binary decisions. The system reaches a fused decision when a specified upper or lower 
threshold has been crossed. We refer to 10 or ||7] for more details. 

We develop an extension of the classical sequential test to cover the following scenario. Only the sensors 
in the first static network are initially active, and the sequential test accepts and combines the fused outputs 
from that network at each point in time. If the system detects an anomaly, it switches over to the second 
static network and continues to pick up and combine measurements in the same manner until a final deci- 
sion has been reached. The motivation for this two-stage setup is that there is typically a cost to activating 
and operating the sensors in the second-stage network, so they are to be switched on only if the first-stage 
sensors decide that there is a good chance of a threat. The two networks do not need to be disjoint and can 
contain some of the same sensors, although possibly with different graph linkages or fusion rules. In prac- 
tical scenarios, the second-stage graph is a superset of the first-stage one that includes additional sensors, 
reflecting the fact that the first-stage sensors continue to collect observations after they trigger any additional 
sensors in the second-stage network. It is also straightforward to add additional stages in the same manner. 
For example, a third stage might correspond to an object being acquired by video before sensors begin to 
collect measurements on it (known as "track before detect"). For clarity, however, we focus on two stages 
in what follows. 

Let H be the object as before, and D n and D' n respectively be the decision outputs of the first and second 
stage fusion networks (at their respective final fusion centers) at time n. Assume that the {D n }\J{D' n } are 
mutually independent. We restrict M = 2 for the rest of the paper, so the static fusion system gives us the 
sequences of 2 x 2 matrices P(D n = -\H) and P'(D' n = - \H) for all times 1 < n < N. We use these inputs 
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to set up the following type of sequential test. We write K, K, K, rio and t?i for respectively the lower 
stopping time, upper stopping time, stopping time, lower threshold and upper threshold for the first stage of 
the test. The thresholds T?o and 77 1 are fixed parameters that control the overall sensitivity of the test, while 
the stopping times are random variables that we will specify below. Similarly, we write f?, K', K', 7]^ and r\[ 
for the corresponding variables for the second stage of the test, where we require that r\' Q < Tjo and T?( > 7]i. 
Let Die = {D n }\< n < k denote the set of decisions of the system's active static fusion network at each time 
n, up to time k. For any sequence of decision possibilities d k = {d n }\< n < k , we define the likelihood ratio 
recursively by 

r(<\)-( M V\ ) p ( Dn = d ^ H = l A ( FT P'(D'n = dn\H=l) \ 
( k) { L\ m> = d n \H = 0)) \ n J { l )+1 P'(D' n =d n \H = 0))> 

where M(d k ) = max{« : n < k,L(d m ) G (t?o, ?]i)Vhi G This allows us to define the stopping times by 

K = min {k : L(D k ) < 7]o,L(D m ) G (j)o,?)i)Vm < k}, 

ke[\,N] 

K = min {k:L(D k ) > T] u L(D m ) G (r)o,T)i)Vm<fe}, 

ke[l,N] 

K = min(^,Z). 

£ = min {k:k > K,L(D k ) < T)^L(D m ) e (^[^m < k}, 

ke[l,N] 

t = min {k:k>K,L(D k ) > T](,L(D m ) G T7O Vm < £}, 

ke[\,N] 

K' = mm(g,K')- 

If any of the above sets is empty, we define the corresponding stopping time to be N + 1 . The test starts with 
the first stage static network P(D n = - \H) and runs like a conventional sequential test until either k = K at 
time k (i.e. when L(Dk) moves outside the region (t]o,T]i)), at which point it switches to the second stage 
network, or until k = N + 1, when it is forced to stop and make a decision by comparing L(D^) to the ge- 
ometric midpoint ^/ttoTTT- If the second stage is triggered, the test continues running with the second stage 
network P'(D' n = -\H) and stops with a final decision when K' = k or K' = N + 1. The first stage's result 
(representing an initial decision) affects whether the second stage starts below j]q or above 771. We want to 
find the statistics of K and K' and the detection and false alarm probabilities of the first stage at each time k, 
denoted Pp and P|, and of the entire test, Pfc' and 

We describe a simple, deterministic algorithm to compute these quantities for incoming, fused sensor mea- 
surements from the two static networks. Let G k be the event {K > k}, i.e. that the first stage was still running 
at time k - 1. We can expand P(K = k\H, G k ) for all 1 <k<Nby writing 

P(K = k\H,G k ) = / J (L(D fc )>7] 1 |L(D m )G(7]o,T7i),l<m<A:-l,^) 

k 

I l\P(D n = d n \H), (3.1) 

A k eA k {7] ,r]i)n=l 



where 



Afe(J7o,f?i) = {djt G {0, 1}* : L(d k ) > r?i,L(d m ) G (?]o, ?]i), 1 < m < k- 1}. 



We can express P(K = k\H) and the k = N+ 1 cases in a similar manner. The number of terms in Ajt(T7o, T71 ) 
generally grows exponentially, but the sum can be computed iteratively by keeping track of likelihood sets 
Jz?£ over time, where each J£ k consists of elements t = (^1,^2) G M 2 . For k = 1, we set ££\ := {P(D\ = 



5 



d\ \H)}. For each k > 1, we find the likelihoods of all possible sample paths, := {P(D k = d k \H) x I : £ £ 
.Sfi-i}, where x denotes the outer product between vectors in M 2 . We then compute 

P(K = k\H,G k ) = £ 

£eJ%/ 2 A'i>J7i 

P(K = k\H,G k ) = £ 

store the likelihoods of paths that escaped ^( k := {£ £ J% : ^2/^1 (HO) ni)}> keep the remaining likelihoods 
Jz?& + i := J£ k \^ k for the next step, and increment k. Note that at each time k, we only need to keep J£ k and 
Jz?&_i in memory. At & = N + 1, any likelihoods J£n+\ still left are summed over in a similar manner. From 
this, we can find 

P(K = k\H,G k )+P{K = k\H,G k ), 

k 

Y,P(K = m\H = l,G k ), 

m=l 
k 

Y J P(K = m\H = 0,G k ). 

m=\ 

In the same way, we can calculate the second stage probabilities. For example, with G' k = {K' < k}, we have 

/M{A k ) \ / k 

P{K' = k\H,G' k ) = £ [Y[P{D n = d n \H)\[ n P'(D' n =d n \H) 

These sums are evaluated in the same way as the first stage probabilities by keeping sets of likelihoods J£' k 
in memory, with the only differences being that for each k < N, we set 5£i := 5£L\^j%\ after computing J£! 
as before (adding paths that cross over between stages), and for k = N+ 1, we set ££' N+X := U-^am-i- 
Finally, we can calculate 

N+l 

E(K\H) = Y, kP ( K = k \ H )> 
k=i 

and any other statistics of K and K' can be found in the same manner. 

This iterative approach can provide a big performance improvement over directly computing (13.11 ) in certain 
situations. The likelihood set A k generally grows like 0(R k ) for some R £ [1,2], but in practice, R is fairly 
close to 1 if either P{D n = l\H = 1) is increasing or P(D n = l\H = 0) is decreasing, which physically cor- 
responds to the target moving closer to the sensor network over time. 

We finally remark that the thresholds rjo and T]i are selected in practice using the Wald approximations, 
= j=£ and T]i = for some target probabilities Pp,Pp £ (0, 1). It can be shown that at the mean 

stopping time k = E(K), P^>\- and P| < p- (@, p. 104). The second stage thresholds 77^ and r][ 
can be chosen in a similar manner with some given P^' and Pp '. 

4 Numerical examples 

In this section, we consider a few example scenarios that illustrate different properties of the static and dy- 
namic elements described above. To clarify the discussion, we consider a simple situation with two sensors, 



P{K = k\H) = 
P k 

pk _ 

F — 
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Si and S2, where the first-stage static network includes only Si, with no fusion, and the second-stage net- 
work includes both Si and S2 and combines them at a Bayes fusion center with uniform costs and priors. At 
each time k for 1 < k < N, N = 25, we assume the sensor Sy has false alarm and detection probabilities of 
0.5 — Aj —Bjk and 0.5 +Aj+Bjk respectively, where the Aj and Bj are some fixed constants and j G {1,2}. 
This setup loosely models a target object moving over time at a constant speed, with only Si being initially 
active and triggering S2 as needed. The thresholds are set using the Wald approximations as described in 
Section [3] 

The sequential test statistics are shown in Figures |4~T1 and FOl for various choices of the above parameters. 
The first scenario corresponds to the target moving closer to both sensors over time, while the second one 
describes a situation where the target maintains a fixed distance from Si, but moves towards S2. In both 
cases, the first-stage thresholds are set up to be modest targets, so that Si quickly makes its preliminary 
decision and activates S2 well before the system reaches its final decision. The stopping time distributions 
generally have large variances and are highly oscillatory in the scenario with stationary sensor statistics, but 
are smooth in the one with a moving target. 

We also calculate estimates of the base R in Section [3l based on the number of sample paths still present 
at the final time N = 25. This gives an indication of how much computation time was saved by culling out 
the sample paths that crossed the thresholds at each time step. We find that R is significantly less than 2, 
especially in the first case, and the running time is several orders of magnitude less than it would be if we 
computed (13.11 ) directly. 




Time (k) 

Figure 4.1: Fusion scenario with {A\,B\ ^2,-82} = {0,0.01,0,0.02} and target probabilities 
{P* F ,P* Dl P F ',P* D '} = {0.3,0.55,0.05,0.99}. 
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Time (k) 

Figure 4.2: Fusion scenario with {A\,B\,A2,B2\ = {0.1,0,0,0.02} and target probabilities 
{P* F ,P* D ,P* F ',P* D '} = {0.2,0.55,0.03,0.999}. 

5 Conclusion 

We have described a general framework for decision-level data fusion performance and tradeoff analysis 
between different sensor configurations. We have discussed an extension of the classical Wald sequential test 
to cover a multiple-stage cueing scenario where the decisions from one sensor network are used to activate 
a second network for a closer look at a target. We have described some numerical examples illustrating the 
behavior of the resulting statistical quantities. These results motivate future work on better characterizing 
the stopping time distributions as well as the dependencies between the first and second stage times. 
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